vqSGD: Vector Quantized Stochastic Gradient Descent

In this work, we present a family of vector quantization schemes vqSGD (Vector-Quantized Stochastic Gradient Descent) that provide an asymptotic reduction in the communication cost with convergence guarantees first-order distributed optimization. process derive following fundamental information theoretic fact: $\Theta \left({\frac {d}{R^{2}}}\right)$ bits are necessary and sufficient (up to additive notation="LaTeX">$O(\log d)$ term) describe unbiased estimator notation="LaTeX">$\hat{\boldsymbol {g}}(\boldsymbol {g})$ for any notation="LaTeX">$\boldsymbol {g}$ notation="LaTeX">$d$ -dimensional unit sphere, under constraint notation="LaTeX">$\| \hat{\boldsymbol {g})\|_{2}\le R$ almost surely, notation="LaTeX">$R > 1$ . particular, consider randomized scheme based on convex hull point set, returns gradient surely bounded norm. We multiple efficient instances our scheme, near optimal, require notation="LaTeX">$o(d)$ at expense tolerable increase error. The obtained using well-known families binary error-correcting codes smooth tradeoff between estimation error quantization. Furthermore, show also offers automatic privacy guarantees.